Various methods and systems for indexing and retrieving video documents are known in the art. In particular, some known techniques are multimodal, i.e., index and retrieve data that is associated with different media. Multimodal techniques are described, for example, by Marsden et al., in “Tools For Searching, Annotation and Analysis of Speech, Music, Film and Video—a Survey,” in Literary & Linguistic Computing, Oxford University Press, volume 22, number 4, November, 2007, pages 469-488, which is incorporated herein by reference. Multimodal indexing of video documents is addressed by Snoeck and Worring, in “A Review on Multimodal Video Indexing,” Proceedings of the 2002 IEEE International Conference on Multimedia and Expo (ICME 2002), Lausanne, Switzerland, volume 2, pages 21-24, which is incorporated herein by reference.
Multimodal retrieval techniques are described, for example, by Amir et al., in “Multimodal Video Search Techniques: Late Fusion of Speech-Based Retrieval and Visual Content-Based Retrieval,” Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '04), Montreal, Canada, May, 2004, volume 3, pages 17-21, which is incorporated herein by reference. The paper describes multimodal systems for ad-hoc search, which use a late fusion of independently-developed speech-based and visual content-based retrieval systems.
Other multimodal retrieval techniques are described by Hoi and Lyu, in “A Multimodal and Multilevel Ranking Framework for Content-Based Video Retrieval,” Proceedings of the 2007 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '07), Honolulu, Hawaii, April, 2007, which is incorporated herein by reference. The paper describes a multimodal and multilevel ranking framework for content-based video retrieval. The framework represents video using graphs, and learns harmonic ranking functions through fusing multimodal resources over the graphs. Multimodal retrieval is also addressed in an evaluation effort entitled TRECVID, which is managed by the National Institute of Standards and Technology (NIST).